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Abstract 


This quasi-experimental study used multiple 
data sources to examine Playworks 
implementation and outcomes across two 
school years (2014-15 and 2015-16) at six 
WCPSS elementary schools. Differences in 
outcomes between Playworks and matched- 
comparison schools were examined. Similar to 
other national studies (Beeker et al., 2012; 
Fortson et al., 2013), the findings of impact are 
mixed. Coaches implemented program 
elements with fidelity, yet WCPSS encountered 
coach turnover as well as inconsistent teacher 
training and lower engagement with students at 
recess than desired. Significant differences 
found in the recess structures and activities 
between Playworks and traditional schools did 
not translate into positive impacts on student 
behavior or school climate. Prior to adopting 
new initiatives, WCPSS should consider the 
following: 1) carefully matching initiatives with 
root causes of school needs and funding 
sources; and 2) increasing the completeness 
and consistency of behavioral data collection 
when reducing students’ negative behavior is 
an expected outcome. Improvements in both 
areas could strengthen the likelihood of impact 
and the ability to measure that impact. 
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Background 


Research has found many benefits of recess including opportunities for physical activity and social 
interaction (Centers for Disease Control and Prevention, 2010; Jarrett, 2013). Physical activity is 
associated with increased cognition, on-task behavior, problem solving, concentration, and 
attentiveness, which may promote academic achievement (Mathematica, 2012). Following the 
research, many pediatricians recommend a daily break of at least 15 minutes during the school day to 
help promote elementary school-aged children’s health and learning (Barros, Silver, & Stein, 2009). 


Despite these benefits, a 2005 National Center for Education Statistics survey (NCES, 2007) found that 
around 85% of U.S. elementary school children have recess, yet the length and frequency of daily 
recess has been declining over the past thirty years. This decline is seen more often among schools 
with high populations of children from low-income families. In their analyses of the same survey, the 
Robert Wood Johnson Foundation (RWJF, 2007) found that large urban schools with high poverty and 
minority student populations tend to have short recesses, and in some cases, no recess at all. 
Additionally, these schools were found to disproportionately use recess deprivation as a form of 
punishment for African-American/Black and Hispanic/Latino male students. Discipline challenges on the 
playground, coupled with increasing pressure to perform on standardized tests, have contributed to 
schools offering fewer opportunities for physical activity during the school day (RWJF, 2007). Other 
research has found that schools have reduced recess time because of problem behavior and bullying on 
the playground (Todd, Haugen, Anderson, & Spriggs, 2002; Craig, Pepler, & Atlas, 2000). 


The North Carolina State Board of Education Heathy Active Children Policy (HRS-E-000) requires 
students in grades K-8 to participate in physical activity as part of the district's physical education 
curriculum. This policy encourages elementary schools to consider providing 150 minutes of physical 
activity a week and middle schools to provide 225 minutes per week including a minimum of physical 
education every other day. It also prohibits taking away physical activity (including physical education) 
or using severe and inappropriate exercise as a form of student punishment. At the district level, 
WCPSS provides regulations and procedures related to physical education as part of Policy 5120 (see 
Appendix B). 


In 2013-14, WCPSS began to implement a program called Playworks at a small number of schools 
serving large proportions of students from economically disadvantaged households. Playworks is a 
national non-profit organization whose mission is to improve the health and well-being of children by 
increasing opportunities for physical activity and safe, meaningful play. Jill Vialet started the program 
in Oakland, California in 1996, after meeting with a principal who expressed concern about the 
frequency of negative incidents occurring during recess (Playworks, 2013). She developed the program 
to support positive play experiences on the playground by assigning each implementing school a full- 
time coach who partners with teachers, principals, and parents to build a culture of play. Playworks 
currently operates in hundreds of low-income elementary schools nationwide. 


The goals of the program are to engage students in physical activity, improve students’ ability to focus 
on class work, develop social skills related to cooperation, decrease behavioral problems, and improve 
school climate (Fortson et al., 2013). Another program goal is that children will develop conflict 
resolution strategies (Playworks, 2013). The primary conflict resolution strategy is “ro-sham-bo,” which 
is a version of “rock-paper-scissors”. This strategy, when applied to situations like whose turn is next in 
a game, provides children with a simple tool for addressing conflict (Playworks, 2013). London, 
Westrich, Stokes-Guinan, and McLaughlin (2015) found that this tool works best for non-emotional 
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issues like negotiating game rules but not serious interpersonal issues, which usually require adult 
intervention. 


Existing research on the Playworks program has typically been conducted during early stages of 
program implementation (primary funders and evaluators include the Robert Wood Johnson 
Foundation, John W. Gardner Center for Youth and Their Communities, and Mathematica Policy 
Research). The various studies on Playworks include: 


1. Studies based on a randomized experiment which demonstrated increases in physical activity for 
girls at Playworks schools (Bleeker, Beyler, James-Burdumy, & Fortson, 2015) and various other 
positive impacts of the program during recess, as discussed below (Bleeker et al., 2012; Fortson et 
al., 2013). 


2. A quasi-experimental study which found higher levels of physical activity and better problem- 
solving skills among students at longer-implementing schools (Madsen, Hicks, & Thompson, 2011). 


3. Descriptive studies of program implementation (London et al., 2015) and outcomes showing, for 
instance, improved school climate at high-implementing schools (Mallonee, London, Stokes- 
Guinan, Westrich, & McLaughlin, 2011). 


In the national, randomized study of Playworks, there is some evidence of positive impacts on teacher 
perceptions of student safety, student engagement in inclusive behavior during recess, student bullying 
and exclusionary behavior during recess, student transitions and attention after recess, and overall 
school climate (Bleeker et al., 2012; Fortson et al., 2013). Students at Playworks schools reported 
better behavior and attention in class after play. These studies showed no negative impacts of the 
program in any assessed domain. Nevertheless, the same research suggests that Playworks had no 
significant impact on any of the following: 


e how well students and teachers treated each other within the school (Bleeker et al., 2012; Fortson 
et al., 2013), 

e teacher reports of student use of positive and encouraging language and aggressive behavior 
(Bleeker et al., 2012; Fortson et al., 2013), 

e student feelings of safety at recess or school, positive peer relationships, and perceptions of overall 
school climate (Bleeker et al., 2012; Fortson et al., 2013), 

e youth development based on eight measures of youth development including problem solving and 
conflict resolution (Bleeker et al., 2012), or 

e negative student behavior during school (U.S. Department of Education, 2013; Bleeker et al., 2012). 
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In 2014-15, DRA was asked to evaluate the implementation and short-term outcomes of the Playworks 
program in the six WCPSS schools. This report presents findings across 2014-15 and 2015-16. As 
shown in Table 1, the data summarized in this report are based on a quasi-experimental design. Cluster 
analysis was used to match each Playworks school (Brentwood, Bugg, Creech Road, Fox Road, Smith, 
and Walnut Creek Elementary Schools) to another school (Lincoln Heights, Cary, East Garner, Dillard 
Drive, North Ridge, and Lynn Road Elementary Schools) based on five measures of student behavior 
from 2013-14. The use of matching generates a counterfactual condition that in theory allows the study 
to assess what would have happened if an individual who was exposed to the treatment (e.g., 
Playworks) had not experienced that treatment. 
Table 1 
Nature of the Data Provided and Valid Uses 


Research Design Conclusions that Can be Drawn 


QO Experimental We can conclude that the program or policy caused changes 
in outcomes because the research design used random 
assignment. 

VI Quasi-Experimental We can reasonably conclude that the program or policy 


caused changes in outcomes because an appropriate 
comparison strategy was used. 


MI Descriptive These designs provide outcome data for the program or 
MI Quantitative policy, but differences cannot be attributed directly to it due 
MI Qualitative to lack of a comparative control group. 


Sources: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014). 


DRA staff used a mixed methods approach to answer several research questions, as shown in Table 2. 
Data collection activities employed in 2014-15 and 2015-16 included a survey of Playworks coaches, a 
teacher survey, recess observations, and analyses of major behavior incident data captured within the 
district’s Student Incident and Referral System (SIRS). In 2015-16, DRA staff conducted a Playworks 
school principal survey and a 4th grade student survey. We also analyzed data from two existing 
sources, the WCPSS Student Engagement Survey (5"" grade students) and the North Carolina Teacher 
Working Conditions Survey. Each data collection activity is described in Appendix C. 


1 In January of 2015, Barwell Road Elementary School started implementing the program; however, it is not 
included as a study school in this report. 
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Were teachers more engaged with students at recess e Recess Observations 
in 2015-16 than in 2014-15? 


Table 2 
Data Sources 
Research Questions Data Sources 
(2014-15 and 2015-16) 
What training did teachers receive in 2015-16? e =Playworks School Principal Survey 
a How were expectations communicated? 
B Did coaches continue to implement the key program  e  Playworks Coach Survey 
[7 components: recess, class game-time, the Junior e Recess Observations 
= Coach Leadership Program (JCLP), and after-school 
) 
ae 
= 


Was there an increase in students showing character e Playworks Coach Survey 
building—leadership, positive interactions with peers, e Teacher Survey 


and conflict resolution skills? e Recess Observations 
Were there fewer student discipline referrals and e Teacher Survey 
problem behaviors/bullying and improved student e Recess Observations 


5 
ability to focus on classwork: e Student Incident and Referral 


System (SIRS) Data (Major Behavior 
Incidents) 
e 4" Grade Student Survey 
Did school climate improve? e 4 Grade Student Survey 
e WCPSS Student Engagement 
Survey (5'" Grade Students) 
e Teacher Working Conditions Survey 
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Overall, the findings of this study do not support the efficacy of Playworks in attaining the stated 
program goals. Following the national model, within WCPSS the program was expected to promote 
student character building in terms of leadership, positive peer interactions, and conflict resolution 
skills, resulting in fewer behaviors leading to discipline referrals at recess and the classroom. It was 
anticipated that this improved behavior would then increase students’ ability to focus on class work 
and reduce bullying in general, ultimately leading to an improved school climate (see Appendix D). 


Coach implementation of the key program components was strong, although teacher implementation 
was lower than the desired level (see Table 4). There were some issues including coach turnover and 
inconsistent teacher training at the school level. In 2015-16, Playworks teachers were less actively 
involved in recess activities than expected, yet their engagement was still significantly greater than 
teachers at comparison schools. Significant differences in the structures and activities of play between 
Playworks schools compared to more traditional recesses, were not sufficient to result in the intended 
positive impacts on student behavior and school climate (see Table 5). 
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Implementation in 2015-16 compared to 2014-15 
What training did teachers receive? How were expectations communicated? 


In 2014-15, principals gave teachers little information about their role in implementing Playworks, so 
DRA recommended additional teacher training in 2015-16 to better define expectations and increase 
engagement. As such, ina survey, principals of Playworks schools were asked how they explained the 
program to their teachers and communicated their expectations for teacher participation during the 
2015-16 school year (see Appendix E). Four out of the six principals participated in this survey. Their 
responses ranged from simply introducing the coach at the beginning of the year to giving a more 
formal, data-based presentation of the program. Three of the four principals reported that they 
discussed teacher expectations at the training. Additionally, all four communicated with and/or 
received feedback from their coach, primarily via email or face-to-face meetings and discussions. 


Did coaches continue to implement the four key program components? 


According to recess observation data, recess implementation was generally strong in both 2014-15 and 
2015-16 with one exception. In 2014-15, DRA recommended that Playworks offer more creative play 
options at recess; however, this did not occur in 2015-16. Survey results based on 4" grade student 
responses indicate that students at both Playworks and comparison schools enjoyed recess, regardless 
of its structure. 


Based on the survey results from five of the six Playworks Coaches in 2015-16, coaches” implemented 
class game time, Junior Coach Leadership Programs (JCLPs), and after-school leagues (see Appendix F). 
Specific findings include the following: 


e Similar to 2014-15, coaches provided class game time on a rotating schedule, so students had class 
game time every three to four weeks. Coaches mentioned the challenge of scheduling class game 
time with teachers. 

e Coaches led a JCLP, which typically consisted of 15 students at each school. Data from 2014-15 
indicated a need for a more consistent use of Junior Coaches (JCs) as active leaders during recess, 
and results suggest that this occurred. In 2015-16, an increase was observed in the percentage of 
recesses that included at least three recess activities led by JCs (49% during the first year and 64% 
during the second year). According to reports from coaches as well as 23 teachers who had JCs in 
their classes, all JCs facilitated some lower grade level recesses. It was often a challenge to get 
teacher permission for JCs to serve during K-3 recesses, according to coaches. 

e Four of five coaches offered after-school leagues, typically basketball and volleyball. Similar to 
2014-15, some coaches had a difficult time getting students to participate, particularly if 
transportation was not readily available. 


Were teachers more engaged with students at recess in 2015-16 than in 2014-15? 
Based on recess observations, Playworks school teachers showed similar levels of engagement with 
students in 2015-16 compared to 2014-15. As in 2014-15, Playworks coaches were more highly 


engaged with students than the teachers were. However, teachers at Playworks schools were more 


? Three of the six Playworks coaches were first-year coaches in 2015-16. 
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actively engaged in terms of leading games and playing with students than comparison school teachers 
(see Appendix G). 


Table 4 
Implementation Outcomes 


2015-16 Change from 2014-15 to 2015-16 


Outcomes Comparison Schools 


Playworks Schools 


Playworks relative to 
Comparison Schools 


Did teachers at NA Mixed-Evidence NA 
Playworks schools 
receive training and Four out of six principals 
expectation guidelines? responded to the survey and 
three indicated that training 
occurred and expectations 
were discussed in 2015-16. 
Did Playworks coaches NA Yes NA 
continue to implement 
the four key program Five out of six coaches 
components? responded to the survey and 
indicated that program 
implementation continued as 
expected in 2015-16. 
Were teachers more Yes No No 
engaged with students 
during recess? * More teachers in Moderate engagement Low engagement overall 
Playworks than overall which declined with little change over 
comparison schools led time 
games and played with 
students 


Note: * indicates a statistically significance difference between Playworks and comparison schools in 2015-16 at 


the p< .01 level. 
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Outcomes in 2015-16 compared to 2014-15 


Was there an increase in students showing character building—leadership, positive peer 
interactions, and conflict resolution skills? 


There is mixed evidence of students showing leadership during recess (see Appendix H). In 2014-15, 
recess games were led by students (other than JCs) during a significantly higher percentage of recesses 
at Playworks schools than at comparison schools recesses. Observers saw the inverse of this 
relationship in 2015-16, in which students at Playworks schools were seen leading games at a 
significantly lower percentage of recesses than at comparison schools. 


With regard to conflict resolution, observational and survey data point to a greater use of specific 
strategies at Playworks schools, but little to no difference in other positive student interactions as 
shown in the following evidence: 


e The percentage of observed Playworks recesses where students demonstrated conflict resolution 
strategies, such as ro-sham-bo? more than doubled (from 24% to 56%) between 2014-15 and 2015- 
16. Student and teacher survey results also support this finding. Conversely, observed use of 
conflict resolution strategies used at comparison school recesses declined slightly. As such, in 
2015-16, a significantly higher percentage of Playworks students were seen using conflict 
resolution strategies than comparison students. 

e Between 2014-15 and 2015-16, there was a decrease in student displays of positive messages at 
both Playworks and comparison school recesses, although the difference between groups was still 
significant (32.5 percentage points) and favored Playworks schools as it had in 2014-15. There 
continued to be no differences between Playworks and comparison school teacher reports of 
students positively working out problems. 


Were there fewer student discipline referrals and problem behaviors/bullying and improved 
student ability to focus on classwork? 


The Playworks program did not appear to improve student behavior overall (see Appendix |). This 
study found no evidence of fewer discipline referrals at Playworks schools and mixed results for 
problem behaviors/bullying. Measures of student transitions from recess to the classroom also 
indicate that students’ ability to focus on classwork did not improve. 


Although Playworks schools were expected to record minor student incidents to monitor program 
implementation; there was considerable school variation in the follow through. As such, major student 
behavioral incidents, which the district requires schools to report, were analyzed instead. In term of 
discipline referrals, as shown in Table 3, at both Playworks and comparison schools, major incidents 
most commonly occurred in the classroom rather than on the playground, and the rates of incidents 
occurring at both of these locations increased since 2014-15. 


3 Ro-sham-bo is a version of the rock-paper-scissors conflict resolution strategy that Playwork coaches teach and 
promote among students. 
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Table 3 
Playground, Classroom, and Total Major Behavioral Incident Rates, 2014-15 and 2015-16 


2014-15 2015-16 
Incident Rate ‘School Incident Rate : School 
Playground Classroom Total : Membership | Playground Classroom Total :| Membership 
Playworks 14 93 17.2! 3,711 2.8 17.1* 313°} 3,532 
Schools (n=6) \ ! 
Comparison 22 9.3 19.0 ! 3,664 41 15.9 35.2 | 3,636 


Schools (n=6) 


Data source: WCPSS Student Incident and Referral System. 
Note: Incident rates are calculated as (# Incidents at Location/School Membership)*100. * indicates a statistically 
significance difference between Playworks and comparison schools at the p< .01 level. 


In 2015-16, student survey reports of problem behavior were actually greater among Playworks schools 
than comparison schools, whereas observational and teacher survey data indicate no differences as the 
following findings show: 


e Fourth grade Playworks students were significantly more likely to report their engagement in and 
punishment for problem behavior than comparison school students, including being sent to the 
principal’s office for doing something wrong during recess, being sent to the principal’s office for 
bad behavior in the classroom, and having to stay after school or sit by themselves during lunch 
because of bad behavior. 

e Conversely, recess observational data show little overall difference in problem behaviors between 
Playworks and comparison school students. The only difference found was the frequency of peer 
teasing, which occurred considerably less often at Playworks schools than at comparison schools in 
2015-16. 

e Similar to 2014-15, there were no significant differences between the two groups of schools in 
teacher reports of students teasing, arguing, or fighting at recess or in teacher responses of bullying 
(based on students reporting the behavior to them) at Playworks or comparison schools. 


Improved student ability to focus on classwork was not found based on the teacher perception data. 
For instance, compared to 2014-15, teachers within both Playworks and comparison schools reported 
an increase in problematic transitions to learning activities after recess in 2015-16. 


Did school climate improve? 


This report used student and teacher survey data to measure school climate (see Appendix J). Students 
answered questions gauging their feelings about school safety and the extent to which others cared 
about them. Teachers responded to questions about school safety and student conduct. The findings 
reveal mixed evidence of improved school climate: 

e In both 2014-15 and 2015-16, a significantly lower percentage of fifth grade students at Playworks 
schools reported feeling safe at school than comparison school students, although this was not 
found for fourth grade students. 

e Also a significantly lower percentage of Playworks school teachers than comparison school teachers 
agreed that their students follow rules of conduct. Although significantly different in 2014-15, by 
2015-16, both Playworks and comparison school teachers held similarly positive views about the 
safety of their school environment and their school’s atmosphere of trust and respect. 
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Table 5 
Comparative Outcomes 


2015-16 


Playworks relative to 
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Change from 2014-15 to 2015-16 


Playworks Schools 


Comparison 
KYol sTole) [3 


Were there increases in student 
character building in terms of...? 


e Leadership 


e Positive interactions with 
peers 


Comparison Schools 


No 


Mixed-Evidence 
* More students displayed positive 
messages during recess 


No difference in students positively 
working out problems 


Mixed-Evidence 
Games led by Junior 
Coaches increased, 
games led by other 
students decreased, 


Moderate overall, some 
decline 


Yes 
Student-led games 
increased 


Low overall, some 
decline 


e Use of conflict strategies Yes Yes No 
*More students used conflict Low overall, increased Low overall, 
resolution strategies during recess decreased 
Were there fewer student No No No 
discipline referrals? Lower overall rate, yet increased High number, rates High number, rates 
considerably for both groups increased increased 
Greater classroom incident rate 
No difference between playground 
incident rate 
Were there fewer student Mixed-Evidence Mixed-Evidence No 


problem behaviors/bullying? 


*More students sent to the 
principal’s office or stayed after 
school as a behavior consequence 


*Less student isolation and teasing 
at recess 


No difference in bullying overall 


Aggression at recess 
increased, arguing 
decreased 


Little change in overall 
bullying 


Teasing at recess 
and overall bullying 
increased 


Was there improved student 
ability to focus on classwork? 


No 
No difference in student transitions 
after recess 


No 
Problematic student 
transitions after recess 
increased 


No 
Problematic student 
transitions after 
recess increased 


Did school climate improve? 


No 
*Fewer 5""-grade students reported 
feeling safe at school 
*Fewer teachers agreed students 
follow rules 


No difference in other measures 


No 
Positive climate with 
some decline over time 


No 
Positive climate 
with some decline 
over time 


Note: * indicates a statistically significance difference between Playworks and comparison schools in 2015-16 at 


the p< .01 level. 
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Discussion and Recommendations 


Prior studies have suggested that Playworks improves physical activity for girls and impacts teacher, 
but not student, perceptions of recess; however, it has not been shown to positively impact problem 
solving, conflict resolution, or inclusive behavior for students in general (Bleeker et al., 2012; Fortson et 
al., 2013; Madsen et al., 2011). Our study did not yield results that were clearly supportive of the 
program either. Significant differences in the structures, activities, and strategies of play between 
Playworks schools compared to more traditional recesses were apparent, but were not sufficient to 
bring about impact in terms of the key student and school outcomes. 


While this report’s findings pertain to one specific program, the recommendations have broader 
implications for WCPSS programs in terms of effectively aligning interventions with school needs and 
capturing the necessary data to determine effectiveness. 


Recommendation 1: Ensure a tight fit between program interventions and school needs. 


Playworks was created in part to offer students opportunities for positive play experiences. Research 
on the program argues that schools often provide students with no or little time to play because 
negative incidents are often more than twice as likely on the playground compared to in the classroom 
(Craig, Pepler, & Atlas, 2000; Todd, Haugen, Anderson, & Spriggs, 2002). It is assumed that if students 
do not possess the conflict resolution strategies necessary to resolve recess disputes, negative 
incidents may carry over into the classroom and result in the loss of instructional time. 


Our findings suggest that the need for Playworks in the targeted schools may not have been strong. 
Negative incidents in WCPSS elementary schools have historically occurred primarily in the classroom, 
rather than on the playground as described in the studies by Craig, Pepler, & Atlas (2000) and Todd et 
al. (2002). WCPSS also did not specifically rely on school indicators such as inadequate play/recess 
time, extensive reports of negative incidents on playgrounds, and/or high suspension rates to select 
Playworks schools. The use of Title | funds to support Playworks obligated the district to select schools 
for service because of their large economically disadvantaged student populations, rather than their 
student behavioral needs. This meant that there was a suboptimal match between the program goals 
and the schools selected for implementation. 


Moving forward, the district should utilize established processes to determine the root causes of school 
issues so that appropriate interventions are selected. For example, the National Implementation 
Research Network suggests the use of the Hexagon Tool. This tool helps districts evaluate evidence- 
based initiatives by considering needs, fit, resource availability, evidence, readiness for replication, and 
capacity to implement (Blasé, Kiser, & Van Dyke, 2013). 


Recommendation 2: Improve efforts in coach retention and teacher training 


Comparisons of low and high implementing schools are mixed within the research on Playworks. 
Similar to the findings of this report, London et al. (2015) did not find significantly less conflict or 
bullying when comparing Playworks schools with high or low levels of implementation. Conversely, 
Mallonee et al. (2011) found that teachers reported that Playworks contributed to fewer conflicts 
during recess and in the classroom. London et al. (2015) also noted two factors that consistently 
influence Playworks implementation: 1) coaches who are skilled and committed to establishing positive 
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relationships with students, and 2) teachers who receive Playworks training are more likely to buy-in to 
the program and support the coach in implementing the program. Consistency of coaches and teacher 
training was not strong within WCPSS. Coach turnover was an issue, particularly in 2015-16, and 
although Playworks school principals were encouraged to better define teacher expectations and 
increase engagement, teacher training was inconsistent across the schools. As such, WCPSS should 
improve efforts to retain coaches and support teachers’ contributions to program implementation if 
this or similar programs are pursued in the future. 


Recommendation 3: Greater consistency in student behavioral incident data reporting 


We recommend that WCPSS leadership create a more standardized minor and major incident rubric 
with stronger reporting expectations and more school-level accountability across the district. 
Currently, the district requires that schools record major incidents, although schools establish their 
own behavioral expectations and consequences, which can lead to considerable school variation. 
Currently, recording minor incident data is encouraged, but not required. As such, there was no 
opportunity to examine possible effects of Playworks on more minor behavioral incidents, except 
through proxy reports like teacher and student surveys, which are valuable data sources, but may be 
more general than specific behavioral frequency counts. 


This recommendation has implications beyond this evaluation of Playworks. Many interventions within 
the district are implemented with the intention of positively impacting student behavior. Thus, 
analyses of both minor and major behavioral infractions are essential for assessing impact, 
necessitating more consistent and reliable data collection systemwide. 
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Appendix A 
Playworks Program Components 


The Playworks program includes four key components which the Playworks coach is expected to 
implement: structured grade-level activities during recess and class game time, the Junior Coach 
Leadership Program, and after-school sports leagues. Guidelines for the successful implementation of 
each component are described below. 


Recess: During all grade-level recess, Playworks coaches are expected to coordinate the playing of 
recess games and sports as well as to introduce skill-building exercises. The Playworks coach is required 
to be at all regularly scheduled recess periods and serves as a recess facilitator rather than as a recess 
supervisor. Teachers and/or other school staff are expected to provide supervision during all recess 
periods. Adults are encouraged to play alongside children, rather than give instructions for play from 
the sidelines. Pelligrini and Bohn (2005) found that adults tend to inhibit students’ exhibition of 
complex forms of play; whereas, peers facilitate social competence and problem solving. During recess 
activities, it is critical that students take leadership and ownership of the games and adults are 
primarily participants in the games. Playworks does not support the limiting or removal of recess 
privileges for extended periods of time as a method of discipline. Each school is expected to have an 
indoor recess plan for inclement weather days. 


Class Game Time: Playworks coaches are expected to provide class game time during which they teach 
students and teachers the rules, expectations, and skills of the recess games and activities. The 
classroom teacher is also required to be present during each session, which should not occur during 
any regularly scheduled recess period. In order to serve all grade levels, it is typical for class game time 
to occur on a rotating schedule and to last 30-40 minutes. According to London, Mallonee, Stokes- 
Guinan, and Westrich (2010), class game time was the most important and enjoyed aspect of Playworks 
for students and teachers. However, class game time is often difficult to implement due to the 
potential for loss in instructional time. Fourth and fifth grade teachers are least likely to implement this 
component (London et al., 2015). 


Junior Coach Leadership Program (JCLP): Part of being a Playworks coach includes facilitating a Junior 
Coach Leadership Program (JCLP) geared toward fourth and fifth grade students. The purpose of the 
JCLP is to build student leadership and ownership of key school functions. Junior coaches are selected 
in the fourth week of Playworks’ programming through a process that includes student application, 
teacher recommendation, and parent permission. 


e During the school day, junior coaches lead games and activities and help students manage conflicts 
at recess. They also have academic expectations such as making up missed work and maintaining 
good grades. 


e After school, junior coaches are expected to attend weekly trainings held throughout the school 
year based on Playworks JCLP curricula designed to help students learn about healthy nutrition and 
violence prevention and support students in taking charge of their health and the wellbeing of their 
communities (RWFJ, 2007). Playworks coaches lead the trainings, which typically include team- 
building games and skill-building activities. Junior coaches are required to attend the trainings in 
order to serve in a leadership role during recess. 
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After-School Leagues: Playworks coaches organize non-competitive girls’ basketball and co-ed 
volleyball leagues for fourth and fifth grade students. The Playworks Coach is responsible for recruiting 
students and for coaching at least one practice per week, generally at the school site. There are also 
weekly evening games in which students at implementing schools play against each other. Game 
nights are typically held at one of the program schools, necessitating travel for the visiting school. 
Playworks leagues take place over a period of approximately 6-12 weeks during the school year. 
Playworks does not provide any student transportation to or from games or practices. 
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Appendix B 
WCPSS 5120 R&P Healthful Living 


Physical Education 


Students enrolled in kindergarten through eighth grade will participate in physical activity as part of the 
district's physical education program. Physical education courses will follow the North Carolina 
Healthful Living Standard Course of Study. 


Elementary School Physical Education 


A minimum of thirty minutes of moderate to vigorous physical activity shall be provided for all 
elementary school students daily. Each elementary school student shall receive 150 minutes of physical 
activity per week. This may include physical education, recess, or other class activities. Included in this 
physical activity shall be at least one day per week of physical education with a certified physical 
education specialist. This physical activity shall be in addition to classroom instruction in health. 


Middle School Healthful Living 


A minimum of thirty minutes of moderate to vigorous physical activity shall be provided for all middle 
school students daily. This may include physical education, recess, or other class activities. Each middle 
school student shall receive Healthful Living instruction for a combined total of 225 minutes per week 
with a certified health and physical education specialist. 


High School Healthful Living 


Each high school student is required to fulfill one credit of Healthful Living with a certified health and 
physical education specialist for graduation from high school. In addition to the Healthful Living course 
requirement, opportunities shall be provided for continuing health and physical education through 
electives. 


e Healthful Living is not to be withheld as a form of punishment for misbehavior or used as work 
make-up time. 

e Severe exercise may not be used as punishment. 

e Students shall be given fitness assessments as defined by a system-wide plan. Assessment results 
will be used to determine areas of program emphasis. 

e Sufficient opportunities for intramural activities shall be provided for students. 
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Appendix C 
Data Sources 2014-15 and 2015-16 


Playworks Coach Survey 


In April 2015 and 2016, we conducted a short online survey of Playworks coaches. Respondents were 
asked to reflect on the current school year and share their experiences. Five of the six coaches 
responded to the survey in 2015-16, compared to responses from every coach in 2014-15. 


Teacher Survey 


In April 2015 and 2016, we asked teachers at Playworks and comparison schools to participate ina 
short, online survey to gain information about program implementation and perceptions of student 
activities and behaviors. The WCPSS teacher survey questions were based on the survey instrument 
used in the national evaluation study of the Playworks and comparison schools (Bleeker, et al., 2012). 
Permission was obtained to use selected survey questions. All classroom teachers were asked to 
respond to questions that measured their perceptions of students’ behaviors, activities, and use of 
conflict resolution skills during recess as well as questions that gauged their level of support for 
organized plan/activities during recess. Classroom teachers at Playworks schools answered additional 
questions about the Playworks program at their school 


We had usable responses from a total of 138 teachers, 77 at Playworks schools and 61 at comparison 
schools, compared to 251 teachers in 2015-16. In 2015-16, the approximate overall response rate was 
under half (44%) of all K-5 classroom teachers in the study schools compared to 75% in 2014-15. 


Table C-1 
Teacher Survey Reponses by School Type 


AYol YoYo) Im BY7 oY] 2014-15 2015-16 

Playworks 121 77 198 
Comparison 130 61 191 
Total 251 138 389 


Recess Observations 


In collaboration with WCPSS Title | administrators, DRA conducted recess observations at each grade 
level (K-5) at the six Playworks and six comparison schools in 2015-16. The purpose of these 
observations was to gather information about implementation of the program, and also to compare 
recess behaviors between Playworks and comparison schools. In addition, maintaining the structured 
protocol utilized in our 2014-15 observations allowed us to compare data across two years (2014-15 
and 2015-16). As displayed in Table C-2, we observed a total of 212 recesses. In 2014-15, DRA 
administrators conducted two recess observations for each grade at each Playworks and comparison 
school (the first observation in March or April and the second in April or May). In 2015-16, Title | and 
DRA administrators conducted one recess observation for each grade level at the same schools during 
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March or April. Observers received observation protocol training and subsequently observed one 
recess with an expert observer. 


Table C-2 
Number of Recess Observations by School Type 


RYol YoYo) im AY7 oY] 2014-15 2015-16 Ne} e-) | 
Playworks 71 36 107 
Comparison 70 35 105 
Total 141 71 212 


WCPSS Student Incident and Referral System (SIRS) Data 


SIRS contains school reported major and minor student incident data for the district. Major incidents 
are managed by school administrators and minor incidents are managed by teachers. In 2014-15, the 
district mandated the recording of all major incidents in SIRS, even if they did not result in an In-School 
Suspension (ISS), Out-of-School Suspension (OSS), or an Alternative Learning Center (ALC) placement. 
Given the inconsistencies in reporting of minor student behavior incidents addressed in 2014-15, this 
study examines the impact of Playworks on student behavior as measured by reports of major incidents 
in 2015-16. 


Playworks Principal Survey 


In April 2016, we conducted a short online survey of Playworks principals. Respondents were asked to 
reflect on the 2015-16 school year and share their experiences. Four of the six principals at the 
Playworks schools in our study responded to the survey. 


Student Survey (Fourth Grade Students) 


In April 2016, we asked fourth grade teachers at Playworks and comparison schools to administer a 
short, online survey to their students. Respondents were asked questions related to their behavior at 
school, their enjoyment of recess, and their school climate. Students who attend a Playworks school 
were also asked questions related to the program. We had usable responses from a total of 814 fourth 
grade students from six Playworks schools and five comparison schools. One comparison school did 
not participate in the survey. The approximate overall response rate was 67%. 


District and State Surveys (WCPSS Student Engagement Survey and NC Teacher Working 
Conditions Survey) 


School climate was measured with seven items on our student survey of 4" grade students at both 
Playworks and comparison schools. In addition, we analyzed WCPSS Student Engagement Survey (SES) 
results for 2014-15 and 2015-16 that reflect 5‘ grade student responses to the same items. 

In addition, we analyzed NC Teacher Working Conditions (TWC) Survey results from the state 
administered survey in 2013-14 and 2015-16 and the district survey in 2014-15. 
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Appendix D 
Pathway of Change 


DRA Report No. 16.07 


Need: Recess is a less structured time for students to engage in play and physical activity. This structure may also provide an environment for students to engage in behaviors 
that will illicit discipline referrals. Students who receive discipline referrals may experience a loss in instructional time because they are expected to go to the office to have the 
issue addressed. As such, there is a need for greater structured recess activities. 


INPUTS 


STRATEGIES 


Short-Term 


OUTCOMES — IMPACT 
Intermediate 


Long-Term 


Playworks places full- 
time coaches in low- 
income elementary 
schools to provide 
opportunities for 
organized play during 
recess. 


Recess and Class Game Time: 
Coaches set-up structured 
activities by grade level during 
recess and class game time. 
Coaches model positive play for 
students to emulate, provide 
positive conflict resolutions, and 
act as role models for teachers 
who may participate in or 
monitor the activities. 


Junior Coach Leadership 
Program: 4°" and 5" grade 
students can participate. 
Coaches train these students in 
leadership and conflict 
resolution so the students can 
act as role models and 
facilitators to lower-grade 
students during recess. 


After-school leagues: 

Coaches provide opportunities 
for 4° and 5" grade students to 
participate in co-ed volleyball 
and girls’ basketball skill- 
building leagues. 


Primary short-term 

e Greater student engagement 
in physical activity 

e = Positive social development 
such as character building in 
terms of positive role model, 
leadership, cooperation, and 
conflict resolution skills. 


Secondary short-term 

e = Positive play and conflict 
resolution will result in fewer 
behaviors that lead to 
discipline referrals. 


Primary Intermediate: 
e Fewer discipline referrals (at 
recess and in the classroom) 


Secondary Intermediate 

e Improve students’ ability to 
focus on class work 

e_—_ Less bullying 


e Improved school climate 
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Appendix E 
What training did teachers receive in 2015-16? How were expectations communicated? 


Of the six principals at Playworks schools, two were new to the school in 2015-16. Principals’ responses 
to how they explained the Playworks program to teachers and communicated their expectations for 
teacher participation ranged from providing very little information to a more formal, data-based 
presentation. Individual principals reported that they... 


e “Didn’t introduce the program (because school had it the prior year), but did introduce the coach. 
Teachers already had a good understanding of the expectations.” This principal also mentioned 
wanting to have a “Playworks Kick-Off” for staff and students at the beginning of the 2016-17 
school year, with an introductory component for new hires and new students. 

e Discussed expectations at a staff meeting at the beginning of the year. 

e Hada meeting before the start of the school year in which teachers were introduced to the 
program, participated in activities, and discussed expectations such as their participation in daily 
recess activities. 

e Explained the details of the Playworks program, shared recess observational data for 2014-15, 
provided by DRA, and used it to support expectations for 2015-16, and provided examples of 
specific activities teachers should engage in during recess each day. 


Principals said they monitored teacher involvement by conducting unplanned recess observations or 
randomly participating in Playworks activities with students. Three of the four principals reported 
giving teachers feedback about what they observed and reminders of expectations through various 
means such as conversations and emails, staff meeting announcements, or weekly memos. Each of the 
four principals said that they communicate with and/or receive feedback from their coach. Three 
specified that they do so via both email and face-to-face meetings, while the other mentioned having 
periodic discussions with the coach. 


Principals described the services coaches provide at their schools. 


e Three principals mentioned how helpful it is to teachers to have a coach who sets up and organizes 
recess activities, providing a more structured recess for students. 

e = Three principals spoke about the coach organizing and leading the after-school leagues. 

e Two principals referred to the coaches’ work with the JCLP in developing student leaders and how 
it would be more difficult to have this program without a Playworks coach. 


Principals also offered different ideas for changes and improvements for implementing Playworks in 
2016-17. Two principals would like to establish weekly meetings with the coach. Other ideas from 
individual principals included: 


e Having the Playworks coach work with teachers to bring Playworks elements into the classroom 
(like norms and routines of behavior and teamwork) 

e Increasing the use of conflict resolution strategies 

e Continuing efforts to encourage teacher involvement in the program 
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Appendix F 
Did coaches continue to implement the key program components: recess, class game-time, 
the Junior Coach Leadership Program (JCLP), and after-school leagues? 


Similar to 2014-15, coaches implemented class game time, the Junior Coach Leadership Program, and 
after-school leagues. It is worth noting that three of the five coaches were first-year coaches at their 
school, with one hired late in the school year after the previous coach left the position. 


Recess: During the 2015-16 year of implementation, 100% of observed Playworks recesses featured a 
variety of games that were set up prior to students’ arrival, marked play boundaries, and an adequate 
amount of play equipment for students (Table F-1). Playworks coaches were observed reviewing recess 
rules at a lower percentage of recesses in 2015-16 than in 2014-15. The percentage point difference 
between these recess structures at Playworks and comparison schools remained fairly consistent 
across years, with more frequent observations of the structures seen during recesses at Playworks 
school. 


There was a notable change in 2015-16 in the variety of sports and games that were offered to 
students during recess at the comparison schools. Between the Playworks and comparison schools, 
there was a 41 percentage point difference in the first year and a three percentage point difference in 
the second year. This decline in difference appears to be a result of a greater variety sport and game 
offerings at the comparison schools. 


Table F-1 
Recess Observation Results: Percentage of Recesses with Observed Structures 


2014-15 2015-16 
Structures Playworks Comparison Playworks Comparison 
RY sYofe) | RYol sToXe) | Difference RYol sfoXe) | RY ol YoYo) | Difference 

Recesses Recesses Recesses Recesses 
Variety of 
sports/games to 100.0 58.6 41.4** 100.0 97.1 2.9 
choose 
Play boundaries a * 
saareed 98.6 68.6 30.0 100.0 71.4 28.6 
Indoor recess plan 98.6 58.6 40.0** 100.0 68.6 31.4** 
ere ae 97.2 58.6 38.6** 100.0 74.3 25.7% 
Recess rules 
reviewed prior to 97.2 10.0 87.2** 75.0 5.7 69.3** 
play 


Note: Observed behavior across all observed grade-level recesses. Asterisks denote statistically significant 
differences between observed behaviors of teachers at Playworks and comparison schools. * p < .01; ** p< .001 
level. Difference reflects a percentage point increase within implementation year for Playworks schools relative to 
comparison schools. Substantial differences across years are indicated in bold. 
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Fourth grade students at Playworks and comparison schools did not differ in significant ways with 
regard to their enjoyment of recess. Nearly all students at both types of schools reported that they like 
and look forward to recess (97.3% and 96% at Playworks schools and 95.6% and 96.7% at comparison 
schools). Students at Playworks were significantly more likely than their comparison schools 
counterparts to report that they play their choice of game during recess (57.9% of Playworks students 
reported doing this a lot compared to 45.4% of their counterparts). They were also significantly more 
likely to report that they play a game that adults want them to play during recess (19.7% of Playworks 
students reported this happening a lot compared to 10.5% of their counterparts). These results suggest 
that students at Playworks schools tend to play more games in general when compared to students at 
comparison schools. 
Table F-2 
Student Survey: Student Activities 


Average and Standard Deviation 


How often do you do Playwork School Comparison School Difference 
the following... Students Students 
Play a game that | 1.54 (.57) 1.37 (.63) 0.17*** 


want during recess 


Play a game that 0.83 (73) 0.63 (.67) 0.20*** 
adults want me to 
play during recess 


Note: Asterisks denote statistically significant differences between Playworks and comparison schools. 
*** 9 < 001 level. Scale ranges from O-Never, 1-Sometimes, 2-A Lot. 


Table F-3 
Student Survey: Student Activities 


Playworks Comparison | Playworks Comparison 


3.6 8.1 38.5 46.6 


How often do % A Lot 
you do the 
following... 
Play a game that | 
want during 


recess 


Playworks Comparison 
57.9 45.4 


Play a game that 
adults want me 
to play during 
recess 
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Teacher feedback on student activities and behavior during recess was collected from our 2015-16 
teacher survey, and the differences in responses tended to follow the pattern seen in 2014-15. The 


three significant differences between Playworks and comparison schools held relatively constant across 


the years, indicating little change in student activities at either Playworks or comparison school 


recesses. 


e Teachers at comparison schools reported seeing their students playing on recess equipment or 


structures (96.7% and 77.9%, respectively) and participating in imaginary games (77.1% and 53.9%) 


more frequently than Playworks teachers reported seeing these activities. 
e Conversely, Playworks teachers saw students were more frequently engaged in playing sports or 
games with adults, such as the coach or a teacher, than comparison teachers did (62.3% and 


11.5%). 
Table F-4 
Teacher Survey Results: Teacher Reported Perspectives of Student Recess Activities 

Percentage of teachers who 2014-15 2015-16 

reported that they Often/Always ; j 

see their students doing the Playworks Comparison Diff Playworks Comparison Diff 

‘ : Teachers Teachers Teachers Teachers 
following during recess: 
Talking with friends 90.9 96.5 -5.6 na na na 
Playing games or sports with other 
98.2 93.1 5.1 92.2 91.8 0.4 

students 

Playing on recess equipment or 

es aa 78.2 94.4 -16.2* 77.9 96.7 -18.8* 

structures 

Playing games or sports with adults 

. ee P 63.6 21.0 42.6* 62.3 11.5 50.8* 
(including yourself) 

Playing creative/imaginary games 51.9 75.7 -23.8* 53.9 77.1 -23.2* 
Sitting or standing someplace 10.1 13.3 -3.2 8.0 18.0 -10.0 
Playing alone 8.3 6.4 1.9 3.9 4.9 -1.0 


Note: “na” indicates the question was not asked in 2015-16. 
Asterisks denote statistically significant differences between teachers at Playworks schools and comparison 


schools. * p < .01 level. Substantial differences across years are indicated in bold. 


Class Game Time: Similar to 2014-15, all five coaches stated that they provide class game time every 
week using a rotating schedule to serve all students; so ideally, a student would have class game time 
every three to four weeks. Two coaches mentioned that they begin class game time with an ice- 
breaker, followed by a main game. Four of five coaches described class game time as an opportunity 
for students to learn and practice the games and social skills that they can transfer to recess. When 
asked to describe some of the successes of class game time, coaches provided the following responses: 
e Students were able to transfer their knowledge of games and social skills like high-fives and ro- 
sham-bo to recess (four of five coaches) 
e Teachers and students were engaged in the class game time activities (three of five coaches) 


Similar to 2014-15 findings, scheduling class game time continued to be a challenge for coaches. Four 
out of five coaches mentioned scheduling challenges such as having a full 30 minutes, fitting into 
teachers’ instructional schedules, and teachers needing to cancel or reschedule. Two coaches 
mentioned that getting teachers and students involved in class game time was sometimes a challenge. 
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Junior Coach Leadership Program (JCLP): Five of the six Playworks coaches led a JCLP, which typically 
consisted of 15 students at each school. The coaches reported that their junior coaches (JCs) attend 
lower grade-level recesses to serve as facilitators. Three of the five said that JCs do this once or twice a 
week, whereas one coach mentioned that this happens daily, and another coach reported a frequency 
of once or twice a month. Frequency varied across schools: three of the coaches specifically mentioned 
that after-school trainings for JCs occur once a week. The main success of the JCLP that was mentioned 
by three coaches was that JCs develop leadership and social skills. 


In 2015-16, no one mentioned recruiting students into the program as a challenge, yet no one 
mentioned it as a success either. Three coaches mentioned the challenges of either gaining teacher 
permission for JCs to miss instructional time to serve during K-3 recesses or getting JCs to sign up to do 
this. The coach who was recently hired faced the challenge of restarting the JCLP late in the year and 
getting the JCs back on track in terms of training, expectations, and serving at recess. 


Twenty-three out of the 77 teachers at the Playworks schools (30%) had at least one student in their 
class who was a junior coach (JC). Each of the 23 teachers said that the JC serves at lower grade level 
recesses. Similar to the national study, most teachers gave positive feedback about the JCs. Three- 
fourths of all teachers agreed that the junior coach helps resolve conflicts at recess. WCPSS JCs appear 
to have been more likely to facilitate conflict resolutions yet less likely to teach other students games 
than national study findings (Bleeker et al., 2015). 


Table F-5 

Playworks Teachers Feedback on Their Junior Coach 
Percentage of teachers in agreement that their WCPSS Study National Study 
Playworks junior coach (N=23) (N=106) 
Serves at a lower grade-level recess 100 na 
Helps resolve conflicts at recess 77.3 66.7 
Teaches other students games at recess 69.6 80.2 
Is a good role model 69.6 74.4 
Has reduced own incidents of conflict with others 65.2 64.4 


Note: “na” indicates the question was not asked in the National Study (Bleeker et al., 2015). Tests of 
significance were not conducted. 


After-School Leagues: Four of five coaches offered after-school leagues, which were typically 
basketball and volleyball leagues. A couple of coaches also offered soccer leagues and one offered 
football. The coach who was hired late in the school year did not offer any leagues, and did not 
mention if the previous coach had done so. Coaches mentioned that they recruited students and led 
practices and league game nights. 


Individual coaches mentioned that the leagues were successful because of parent and school support 
and because they promoted student enjoyment of the sport and gave students a chance to learn skills 
and to be part of a team. Two coaches mentioned that securing transportation for students after 
practice and on game nights was a challenge and another two said getting students to show up to 
practices and games was a challenge (this may or may not be related to transportation issues). 
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Appendix G 
Were teachers more engaged with students at recess in 2015-16 than in 2014-15? 


As shown in Table G-1, overall, Playworks teachers were not more engaged with students during recess 
in 2015-16 than in 2014-15; though their engagement was still higher than what comparison school 
teachers exhibited. Recess observations show the following results: 


e Alower percentage of Playworks teachers were observed modeling positive messages in 2015-16 
compared to the previous year, resulting in no difference in this behavior compared to teachers at 
comparison schools. 

e Relative to comparison schools, teachers at Playworks schools were still more actively engaged 
with students during recess, based on observations of teachers supervising or leading a game (even 
though observations of this behavior declined among Playworks teachers) or playing with students 
(observations of this behavior declined among Playworks teachers and increased among 
comparison school teachers). 

e No observed difference in Playworks and comparison school teachers walking around and 
monitoring students during recess persisted in 2015-16, although this behavior increased slightly 
among Playworks teachers. 


Table G-1 
Recess Observation Results: Percentage of Teachers Engaged in Behaviors across Recesses 


2014-15 2015-16 
Playworks Comparison Difference Playworks Comparison | Difference 
Teachers Teachers Teachers Teachers 
PUPEMMSInEMEaCINE S| og 9.8 32.1** 30.4 4.3 26.1** 
game 
Playing with students 29.5 9.0 20.4** 32.9 14.8 18.1* 
MoHeline Rose 30.7 14.0 16.7* 19.5 15.8 3.7 
messages 
Encouraging student 
aces 33.2 21.8 11.4 20.6 16.9 3.7 
participation 
ae ieerecne 48.1 53.8 ae, 60.9 51.4 9.7 
monitoring students 


Note: Average percentage of teachers engaged in behaviors across all observed grade-level recesses. Teachers 
may have been observed engaging in multiple behaviors. Asterisks denote statistically significant differences 
between observed behaviors of teachers at Playworks and comparison schools. * p< .01 level, ** p< .001 level. 
Difference reflects percentage point increase or decrease within year for Playworks relative to comparison 
schools. Substantial differences across years are indicated in bold. 
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During 2015-16, coaches engaged in a variety of behaviors with students during nearly all recesses 
(Table G-2). The largest increase was observed in the percentage of recesses where coaches monitored 
students, which increased by almost a third from the first to second year. 


Table G-2 
Recess Observation Results: Percentage of Recesses Including 
Observed Behaviors of the Coach 


2014-15 2015-16 Difference 
EAN e) <3 aE AW el <3 
(@oF-Tol) Coach 
Playing with students 88.7 97.2 8.5 
Encouraging student participation 84.5 97.2 12.7 
Modeling positive messages 80.3 97.2 16.9 
Supervising/leading a game 80.3 97.2 16.9 
Monitoring students 70.4 94.4 24.0* 


Note: Observed behavior of coaches across all observed grade-level recesses. * p < .01. 
Difference reflects the percentage point increase from 2014-15 to 2015-16. 
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Appendix H 
Was there an increase in students showing character building—leadership, positive peer 
interactions, and conflict resolution skills? 


We measured leadership based on our observations of students leading games during recess. In 2014- 
15, recess games were student-led during a significantly higher percentage of recesses at Playworks 
schools than comparison schools. Observers saw the inverse of this relationship in 2015-16 due to a 
large increase in the percentage of recesses with students leading games at comparison schools (50% in 
2014-15 to 97.1% in 2015-16). 
Figure H-1 
Recess Observation Results: Percentage of Recesses with Students Leading At Least 3 Games 
100% 97.1% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% 


m Playworks Recesses 


= Comparison Recesses 


2014-15* 2015-16* 
Note: Asterisks denote statistically significant differences between observed behaviors of students at Playworks and 
comparison schools with the given year. * p< .01 level. 
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In terms of positive peer interactions, in 2014-15 and 2015-16, relative to comparison schools, students 
at Playworks schools were observed displaying positive messages, such as high fives, fist bumps, and 
encouraging language, during a significantly higher percentage of recesses. However, among both 
groups, this behavior was observed at fewer recesses in 2015-16 than in 2014-15. 


Figure H-2 
Recess Observation Results: Percentage of Recesses with Students Displaying Positive Messages 


100% 
90% 
80% 73.2% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


m Playworks Recesses 


m= Comparison Recesses 


2014-15* 2015-16* 


Note: Asterisks denote statistically significant differences between observed behaviors of students at Playworks and 
comparison schools with the given year. * p< .01 level . 


Ro-sham-bo is intended to be the core strategy for resolving conflicts at Playworks schools. As shown 
in Figure H-3, relative to comparison students, in 2015-16, Playworks students were observed 
demonstrating conflict resolution strategies, including but not limited to ro-sham-bo, during a 
significantly higher number of recesses (55.6% compared to 17.1% at comparison schools). This 
difference is a result of the percentage of Playworks recesses where students used conflict resolution 
strategies doubling between 2014-15 and 2015-16. 


Interestingly, we did not find statistically significant differences in teacher reports of students 
often/always positively working out problems together or asking them for help resolving a conflict, as 
shown in Table H-1. 


In 2015-16, we also asked fourth grade students attending Playworks schools how often they use ro- 
sham-bo to resolve conflicts during a normal day at recess. Over half reported that they use ro-sham- 
bo three or more times during recess, and most students (85.5%) reported using it at least once. Both 
years, a great majority of Playworks teachers also agreed that their students used conflict resolution 
skills, like ro-sham-bo, both in class (86.5% in 2014-15 and 82.9% in 2015-16) and during recess (85.6% 
in 2014-15 and 90.8% in 2015-16). 
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Recess Observation Results: Percentage of Recesses with Students Demonstrating Conflict Resolution 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


23.9% 22.9% 


2014-15 


2015-16* 


m Playworks Recesses 


Comparison Recesses 


Note: Asterisks denote statistically significant differences between observed behaviors of students at Playworks and 


comparison schools with the given year. * p< .01 level . 


Table H-1 


Teacher Survey Results: Teacher Reported Perceptions of Student Recess Conflict Resolution 


Percentage of teachers who 2014-15 2015-16 

reported that students ; . 

often/always do the following Playworks Comparison piference Playworks Comparison Biikercnce 
. Teachers Teachers Teachers Teachers 

during recess... 

Positively work out problems 518 A76 A2 559 533 26 

with other students 

Aer Teachiels TOME THEM 36.0 28.0 8.0 355 32.8 0.3 

solve a conflict 


Note: Differences across years were not statistically significant. 
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Appendix | 
Were there fewer student discipline referrals and problem behaviors/bullying and improved 
student ability to focus on classwork? 


Student Discipline Referrals 


Given the inconsistencies in school-level reporting of minor student behavior incidents that we 
encountered, this study examines changes in reports of major student behavior incidents. Beginning in 
the 2014-15 school year, all WCPSS schools were expected to use the Student Incident and Referral 
System (SIRS) to record major incidents, i.e., behavioral infractions resulting in an office discipline 
referral that required administrative action. Similar to 2014-15, in 2015-16 the largest proportion of 
major incidents occurred in the classroom. In 2015-16, Playworks schools had significantly higher rates 
of classroom and total incidents than comparison schools, yet there were large increases within both 
groups compared to 2014-15. 


Table I-1 
Number and Percentage of Major Behavioral Incidents on the Playground and in the Classroom, 
2014-15 and 2015-16 


2014-15 2015-16 
Playground: Classroom ' Total # Playground: Classroom ' Total # 
Incidents Incidents ‘Major Incidents Incidents ' Major 
%of | %of | Incidents # %of | # %of | Incidents 
# Total # Total Total Total 
Playworks 
Schools 51 8.0% : 344 53.8% ' 640 100 9.0% ' 605 54.7% ' 1,107 
(n=6) 3 3 | ! 
Comparison 
Schools 81 11.7% + 340 48.9% | 695 149 11.6% | 577 45.0% + 1,281 
(n=6) ! ! ! ! 


Note: Percentages of playground and classroom incidents are interpreted as the proportion of all major incidents 
that were reported as occurring at those locations. The remaining incidents occurred at other locations such as 
the bus, the cafeteria, the hallway, the restroom, the gym and other school areas. 
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Tables I-2 and I-3 show the total major incident rates for schools that began Playworks implementation 
in 2014-15*. Although the Playworks schools yielded a higher pre-implementation major incident rate 
than comparison schools, by 2015-16, the group had a lower major incident rate than comparison 
schools. The rates for both groups increased considerably after two years of implementation. 


Table I-2 
Major Behavioral Incident Rates Before Implementation, 2013-14 
Pre-Implementation 
2013-14 
Major 
Incident : Total# : School 
Rate _: Incidents ' Membership 
Playworks Schools (n=3) 8.0 (155 | 1,934 
Comparison Schools (n=3) 5.0 86 1,705 


Note: The incident rate is calculated as (# Incidents/School Membership)* 100 


Table I-3 
Major Behavioral Incident Rates After Implementation, 2014-15 and 2015-16 


Post-Implementation 
Year 1 (2014-15) Year 2 (2015-16) 

Major : Major 

Incident ; Total# : School Incident ; Total# ; School 

Rate : Incidents : Membership Rate* < Incidents : Membership 
PIaYWOFKS SENOS | “-aqg. 4 O50. |: 1838 26.7 ! 476 |} 41,783 
(n=3) 
baneon 154 : 260 : 1,718 324 } 545 | 1,683 
Schools (n=3) 


Note: The incident rate is calculated as (# Incidents/School Membership)*100. Asterisk denotes 
statistically significant differences between the major incident rate at Playworks and comparison schools 
in 2014-15. * p< .01 level. 


4 Brentwood, Fox Road, and Walnut Creek implemented in 2014-15 and 2015-16. We selected these schools for 
our pre- to post-implementation analysis because we had pre-implementation incident data in 2013-14. 
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Student Problem Behavior and Bullying 
During 2015-16, relative to comparison students, Playworks students were observed: 


e teasing peers at significantly fewer recesses (3% for Playworks schools and 26% for comparison 
schools). The percentage of Playworks recesses that included student teasing decreased somewhat 
across years; the percentage of recesses at comparison schools that included teasing increased by 
more than eight fold from the first to second year. 

e arguing with peers at slightly fewer recesses (27% during the first year to 22% during the second 
year), yet Playworks recesses were not statistically different from comparison recesses during 
either year. 

e showing aggression toward peers at more recesses, although the difference was not significant. 
This behavior increased for both groups across years (13.7 percentage point increase for Playworks 
schools and 5.7 for comparison schools). 


Table I-4 
Recess Observation Results: Percentage of Recesses Including Observed Behaviors by Students 


2014-15 2015-16 
; Playworks Comparison Playworks | Comparison 
Student Behavior , ; 
RYol YoYo) | RYol YoYo) | Difference RYol YoY) | RYol sYoXe) | Difference 
Recesses Recesses Recesses Recesses 
Tease peers 5.6 2.9 2.8 2.8 25.7 -22.9* 
Argue with peers 26.8 35.7 -9.0 22.2 20.0 2.2 


Show aggression 
14.1 11.4 2.7 27.8 17.1 10.7 
toward peers 


Note: Observed behaviors across all observed grade-level recesses. Average percentage of students engaged in 
behavior across all recesses. Asterisks denote statistically significant differences between observed behaviors of 
students at Playworks and comparison schools. * p < .01 level; ** < .001 level. Difference reflects percentage 
point increase or decrease within year for Playworks relative to comparison schools. Substantial differences 
across years are indicated in bold. 
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Both years had no statistically significant differences in teacher reports of students often/always 
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arguing or teasing or fighting at recess. Interestingly, Table I-9 (item #3) shows that significantly higher 
percentages of Playworks teachers agreed that arguing/fighting or conflict are reduced when student 


have structured recess. 


Table I-5 
Teacher Survey Results: Teacher Reported Perceptions of Student Recess Behavior 


Percentage of teachers who 2014-15 2015-16 

reported that students . j 

Often/Always do the following Playworks Comparison piterence Playworks Comparison Difference 
: Teachers Teachers Teachers Teachers 

during recess... 

Tease other students about not 

being good at games/sports 6.3 4.2 2.1 5.2 6.6 -1.4 

Get into an argument with other 

students 18.9 14.6 4.3 15.6 24.6 -9.0 

Fight or hit other students 5.4 2.8 2.6 4.0 8.2 -4.2 


Note: Differences across years were not statistically significant. 
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Teacher Reports of Student Bullying 


A student bullying/exclusion scale score was calculated by averaging responses to six survey items 
representing various aspects of behaviors leading to student isolation or exclusion. The average 
differences between groups were not statistically significant during either year, although both appear 
to be trending upward. 


Table H-6 shows the percentage of teachers who indicated that students reported each of the six 
measures of bullying and exclusion and the scale score. Findings include the following: 


e = Asignificantly lower percentage of teachers at Playworks schools than comparison schools 
reported student isolation from their peers. 

e =Additionally, in 2015-16 there was no longer a significant difference between teacher reports of 
students’ fear of being bullied at school, due to an increase in reports from comparison teachers. 


Table I-6 
Teacher Survey Results: Percentage of Teachers Reporting Student Bullying/Exclusion 
2014-15 2015-16 
In the past 30 days, how often have your Once or twice Once or twice 
students reported to you that they have in the past 30 days in the past 30 days 
been: Playworks | Comparison Diff Playworks | Comparison Diff 
Teachers Teachers Teachers Teachers 
Hit or pushed by another student during 57.4 573 01 701 54.1 16.0 
recess 
Name-called during recess 58.3 60.4 -2.1 62.3 59.0 3.3 
Isolated from their normal peer group 393 49.7 -10.4* 33.8 50.8 -17.0* 
during recess 
Bossed or coerced into doing something 27.1 347 76 23.4 295 61 
they didn't want to do at recess 
Afraid to go to recess for fear of being 33 4.9 ae 11.7 re 6.8 
bullied 
Afraid to come to school for fear of being 14.7 4.9 9.8% 10.4 115 Ala 
bullied 
A A 
Student Bullying/Exclusion Scale Score Total Scale Average hag Total Scale Average ae 
WCPSS (N=251) 0.63 0.59 0.04 0.68 0.81 0.13 
National Study (N=245) 0.6 1.0 -0.5* na na na 


Note: The scale in the WCPSS study included responses from 0-3 with 0 representing “never” and 3 corresponding 
to “3 or more times.” The scale in the national study ranged from 0-3 with O representing “never” and 3 
corresponding to “5 or more times.” A higher average scale score indicates a higher frequency of 
bullying/exclusionary behavior. Asterisks denote statistically significant differences between teacher reports at 
Playworks and comparison schools. * p < .05 level. Note: “na” indicates that the national study (Bleeker et al., 
2015) was conducted in 2014-15 only. 


36 


Playworks 2014-15 and 2015-16 DRA Report No. 16.07 


Student Reports of Being Disciplined for Problem Behavior 


Six items on the student survey comprised a measure of the punishments students received for their 
problem behavior. We examined averages and standard deviations for each item and also calculated a 
Student Behavior Scale Score, which is an average across the six items for each student. Higher 
averages suggest that students experienced the behavior more frequently throughout the year. 


Students predominately reported that these behaviors never occurred or if they did occur, they did so 
only once or twice during the school year. Still, the scale score means for Playworks and comparison 
schools were statistically different indicating that Playworks students reported that the frequency of 
problem behavior was significantly higher than reports by comparison school students. This finding 
differs from the national study (Bleeker et al., 2015) in which no significant mean difference was found. 


Table I-7 
Student Survey: Student Behavior 


Average and Standard Deviation 


During this school year, how Playworks School Comparison School Difference 
often have you Students Students 

Gotten in trouble for something 0.58 (.80) 0.51 (.70) 0.07 
that happened during recess 

Had to sit out at recess for bad 0.38 (.69) 0.32 (.62) 0.07 
behavior 

Been sent to the principal’s office 0.14 (.48) 0.05 (.22) 0.09*** 


for doing something wrong 
during recess 


Been sent to principal’s office for 0.26 (.67) 0.12 (.39) 0.14*** 
bad behavior in the classroom 

Had to stay after school or sit by 0.43 (.76) 0.18 (.46) 0.25*** 
yourself at lunch because of bad 

behavior 

Been suspended from school 0.12 (.49) 0.08 (.38) 0.04 
WCPSS Study Behavior Scale 0.32 (.46) 0.21 (.31) 0.11*** 
Score 


National Study Behavior Scale 


Score (Bleeker et al., 2015) 0.3 0.4 -0.1 


Note: Asterisks denote statistically significant differences between Playworks and comparison schools. 
*** 9 < .001 level. Scale for WCPSS study: O = Never; 1 = Once or Twice; 2 = Three or Four Times; and 3 = Five or 
More Times during the school year. 
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The percentage of teachers within both groups who reported problematic transitions to learning 
activities after recess increased considerably in 2015-16 for both groups. Group differences were not 


statistically significant. 


Table I-8 
Teacher Survey Results: Percent of Teachers Who Reported Student-Transitions 


Think about the most recent school day in 2014-15 2015-16 
which your students participated in 
recess. In the 15 minutes just after recess, Playworks | Comparison ; Playworks | Comparison ; 
wie Aurea Diff Diff 
please indicate which, if any, of the Teachers Teachers Teachers Teachers 
following were true: 
Some students became restless and lost 
. 28.6 27.9 0.7 57.9 65.4 -7.5 
focus on their tasks 
There were incidents of negative behavior 
toward peers (teasing, name-calling, 19.5 16.4 3.1 39.5 38.5 1.0 
aggression, or exclusionary behavior) 
| spent more time than | would have liked 
24.7 18.0 6.7 50.0 42.3 7.7 


redirecting student misbehavior 


Note: Differences across years were not statistically significant. 


As shown in Table I-9, classroom teachers at Playworks and comparison schools were asked to respond 
to eight items to gauge their support for organized play/activities during recess. These activities 

include traditional sports, new games, and creative play with specified rules and designated spaces for 
participation that are taught or led by an adult or students. 


e Similar to 2014-15, Playworks teachers were significantly more supportive of organized play than 
comparison school teachers, although their support appeared to have declined in 2015-16. 

e = Significantly more Playworks teachers agreed or strongly agreed that when students participate in 
organized play they are more likely have increased physical activity levels and to feel included, and 
that fights and conflict in the classroom are reduced and the transition back to class is shortened. 

e Comparison school teachers were overall less positive about organized play and were more 
concerned about students losing time for unstructured play. 
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Table I-9 
Teacher Survey Results: Teacher Support for Organized Play 
2014-15 2015-16 
Percentage of teachers who Playworks Comparison Diff Playworks Comparison Diff 
Agree/Strongly Agree with the Teachers Teachers Teachers Teachers 
following statements... 
1. Participating in organized 91.0 62.6 28.4* 90.8 67.2 23.6* 


play/activities during recess helps 
increase students’ physical 
activity levels. 

2. Students are more likely to feel 93.7 62.3 31.4* 88.2 67.2 21.0* 
included if they participate in 
organized play/activities during 
recess. 

3. When there are organized 83.8 45.3 38.5* 76.6 50.8 25.8* 
play/activities during recess, 
students are less likely to get 
involved in arguments or fights. 

4. Conflict in the classroom is 78.9 40.8 38.1* 75.0 39.4 35.6* 
reduced if students have 
participated in organized 
play/activities during recess. 

5. The transition back to class after 66.7 37.4 29.3* 59.7 41.0 18.7* 
recess is shortened if students 
have participated in organized 
play/activities during recess. 

6. Participating in organized 35.1 62.9 -27.8* 36.9 68.3 -31.4* 
play/activities during recess takes 
away important time that 
students have for unstructured 
play. 

7. Scheduling physical activity 32.4 23.5 8.9 23.4 23.3 0.1 
programs during the school day 
takes away important time that 
students need to focus on 
academic achievement. 

8. It is important for students to 93.7 98.5 1.2 94.8 98.4 -3.6 
have the opportunity to have 
recess during the school day. 


Asterisks denote statistically significant differences between observed behaviors of teachers at Playworks and 
comparison schools. * p < .01 level . 
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Appendix J 
Did school climate improve? 


School climate was measured with seven items on our student survey of 4" grade students at both 
Playworks and comparison schools. We also analyzed WCPSS Student Engagement Survey (SES) results 
for 2014-15 and 2015-16 that reflect 5* grade student responses to the same items. Overall results 
suggest that the school climate between Playworks and comparison schools were not significantly 
different in 2014-15 and 2015-16. 


Table J-1 
Student Survey: Fourth and Fifth Grade Student Survey Results, 2014-15-to 2015-16 


2014-15 2015-16 2015-16 
ce (eT e-lel=) ae (eyelets 4" Grade 


Percentage of Playworks Difference Playworks Difference Playworks Difference 
students agreeing Students ligeyanl Students from Students lige)aa) 
with the following Comparison Comparison Comparison 
statements: Students Students Students 
At my school, teachers 


care about students. “11 


My teachers are there 
for me when | need 
them. 

Most teachers at my 
school are interested in 
me as a person, not 
just as a student. 


| feel safe at school. 


Other students at 
school care about me. 


Students at my school 
are there for me when 
| need them. 

Students here respect 
what | have to say. 


Note: Agreement includes “Strongly Agree” and “Agree” response categories. 
Bold asterisks denote statistically significant differences between teachers at Playworks schools and comparison 
schools * p<.01 level. 
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In addition, we analyzed several items related to school climate from the Teacher Working Conditions 
(TWC) Survey results from the state administered survey in 2013-14 and 2015-16 and the district 
teacher’s survey in 2014-15. In 2013-14, a significantly higher percentage of comparison school 
teachers than Playworks teachers agreed that their students follow rules of conduct. This difference 
persisted in 2015-16. By 2015-16, there were no longer significant differences across the groups in 
terms of teacher agreement about the safety of the school environment and the school’s atmosphere 
of trust and respect. 


Table J-2 
Teacher Survey: North Carolina Teacher Working Conditions (NC TWC) 
and WCPSS Teacher Survey Results, 2013-to 2015-16 


NC TWC Survey WCPSS Teacher Survey NC TWC Survey 
2013-14 2014-15 2015-16 
Playworks Comparison Diff Playworks Comparison Diff Playworks Comparison Diff 
KY al sore) RY al afore) RY sTeXe)| NYel sTefo)| KY al sore) KY ol sfoye)| 


Students at 
this school 
follow rules 
of conduct. 
The faculty 
work ina 
school 
environment 
that is safe. 
There is an 
atmosphere 
of trust and 
mutual 
respect in 
this school. 


Which aspect of your teaching conditions is most important to you in 
promoting student learning? 


Managing 

student 

conduct 

Total 

Responses 
Note: Agreement includes “Strongly Agree” and “Agree” response categories. 
Bold asterisks denote statistically significant differences between teachers at Playworks schools and comparison 
schools * p<.01 level ** p<.001 level. “na” indicates the question was not asked on the WCPSS TWC Survey. 
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